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ABSTRACT 



The student affairs researcher often assists student affairs 



directors and professionals to accomplish their program assessment goals. 

This study identifies the information, consultation, and training needs in 
research and assessment of student affairs professionals at a research 
institution. The paper also summarizes the problems and possibilities of 
using Web-based surveys, and describes how the results might impact the 
research office in advancing the program assessment goals of the division. Of 
98 full-time student affairs personnel invited to participate, 26 
mid-managers and 42 professional staff members responded to a Web survey 
instrument developed for the study. Results show that most student affairs 
professionals report a basic level of research and assessment expertise. 
Although mid-managers and staff members had common needs, each group had 
distinctive interests. Mid-managers were more interested in accountability 
measures and the design of assessment projects. Staff members favored more 
specific service-oriented measurement tools. Student affairs personnel used 
e-mail and were positioned to respond to activities presented on the 
Internet. Mid-managers were especially concerned about how to integrate 
technology into program services, but staff members were mainly concerned 
about adequate training in the use of new software and technology. The paper 
discusses the implications of survey findings for student affairs 
professionals and student affairs research offices, which are often 
one-person offices with minimal budgets and multiple responsibilities. 
(Contains 8 tables and 17 references.) (SLD) 
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A Web-Based Measurement of the Assessment Needs of 
Student Affairs Professionals and the Role of the Student Affairs Researcher 

Abstract 

The student affairs researcher often assists student affairs directors and professional 
staff to accomplish their program assessment goals. This study (1) identifies the 
information, consultation and training needs in research and assessment of student 
affairs professionals at a research institution, (2) summarizes the problems and 
possibilities in using web-based surveys and (3) describes how the results might impact 
the research office in advancing the program assessment goals of the division. 
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A Web-Based Measurement of the Assessment Needs of 
Student Affairs Professionals and the Role of the Student Affairs Researcher 

Introduction 

Institutional researchers in student affairs, regardless of size of the institution, 
often conduct institutional and student affairs studies. Many conduct institutional 
student outcome studies (Hyman, Beeler, and Benedict, 1994) employing instruments 
such as the Cooperative Institutional Research Program (CIRP), College Student 
Experiences Questionnaire (CSEQ) or National Study of Student Engagement (NSSE). 
They also engage in a range of other activities typical of an I R office (Malaney, 1999; 
Malaney, 1993; Thurman and Malaney, 1989). 

Administrators in student affairs recognize student-oriented research and 
assessment as an important function (Erwin, 1989) and as one way to demonstrate the 
value and accountability of their programs and services (Hanson, 1990; Malaney, 1993; 
Schuh and Upcraft, 2001; Sorochty, 1991; Upcraft and Schuh, 1996; Winston and 
Miller, 1994). 

Student affairs directors and professional staff are expected to perform a range 
of assessment tasks. They are expected to select or design appropriate instruments to 
measure their program goals, collect pertinent developmental and environmental data, 
count and monitor contacts, process and analyze data, utilize results to improve 
programs, determine program demand/use, effectiveness and efficiency, and meet 
internal and external standards. These are only a few of the relevant tasks. 
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Similarly, student affairs researchers are expected to work with student affairs 
mid-managers in conducting program evaluations and other activities (Moxley, 1988). 
However, student affairs mid-managers are often hired for their leadership, student 
contact, communication and other skills rather than for assessment competencies 
(Gordon, Strode and Mann, 1993). Too often the training and development needs of 
mid-managers are based on administrative opinions rather than on results derived from 
research, such as a written survey or an interview (Upcraft, 1988). Ways of 
systematically and easily measuring the assessment needs of student affairs 
professionals are needed to facilitate their staff development, to assist in achieving 
program assessment goals and to enhance researcher-personnel collaborative efforts. 

The purpose of this study is to: (1) identify the information, consultative and 
training needs in research and assessment of student affairs mid-managers and 
professional staff at a research institution, (2) summarize the problems and possibilities 
in using a web-based survey and (3) describe how the results might impact the student 
affairs research office in advancing the program assessment goals of the division. 

The student affairs research office at this institution conducts entering and 
undergraduate student surveys, assists in program assessment activities, participates 
in accreditation reviews, performs special studies, handles data queries, disseminates 
reports, participates in the Institutional Review Board and provides consultation and 
training to student affairs personnel. It operates independently from but cooperatively 
with a fully staffed university system Institutional Research Office. 
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Method 

Respondents 

Of 98 full-time student affairs personnel, 26 mid-managers and 42 professional 
staff participated in this study (Mills, 2000, Scott, 2000). Return rates were 100% for 
mid-managers and 58% for staff for an overall return rate of 69%. Mid-managers 
include directors and coordinators of programs; staff include faculty specialists and 
administrative/professional/technical (APT) staff. Their e-mail addresses, obtained 
through the division directory, were tested and updated. Participants were informed 
about the survey through announcements at various meetings. Announcements were 
followed by an e-mail invitation to participate in the survey. That e-mail was linked to 
the web survey. One staff member with no electronic address was sent a hard copy of 
the survey. Another visually challenged director was interviewed by telephone. 

The Survey Instrument 

The Web survey instrument (OSA Assessment Inventory), developed during the 
summer of 2000, contains five parts: respondent background information; interest in 
assessment information, consultation and training; technology and on-line searches; 
statistical analysis; and scheduling and planning. Items were developed after 
considering the following: current and future assessment issues of the institution, 
interests of student affairs personnel, topics in national and regional assessment 
workshops, a 1992 needs assessment administered to the division by the investigator; 
and, web survey design considerations (Dillman, 2000). 

Respondents took about ten minutes to complete the 95-item survey. Survey 
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items were composed of one choice items, multi choice items, repeated scale items, fill 
in the blanks, fill in the numbers and short essays. A paper version of the survey 
provided an option to a respondent who was visually challenged and another who 
lacked an internet connection. 

Data Analysis 

Results were processed by downloading the data file from the university server 
to SPSS (Statistical Package for the Social Sciences) for analysis on an office PC. 

Data were also captured in e-mail format, monitored and used to create follow-up 
reminder lists. The analyses of results include descriptive statistics (percentages and 
mean) and chi-square using an alpha level of .05. 

Results 

Respondent Background 

Respondents averaged eight years of experience in their current or closely 
related position in student affairs. Length of experience ranged from .5 to 30 years. 
Most respondents (40%) were from small programs composed of 3-6 members. 
Although the majority (53%) described their assessment experience as basic, two-fifths 
(40%) of mid-managers characterized their expertise as intermediate. 

Interest in Assessment Topics 

This section was a focal point of the survey. Mid-managers and staff were 
asked to respond to 31 assessment topics. They rated these topics on a scale of 1 = 
Very Interested, 2 = Somewhat Interested and 3 = Not Interested in obtaining 
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information, consultation or training in these topics. See Table 1 for a list of the 31 
topics. 

[Place Table 1 about here.] 

The results of a chi square analysis indicated that there were significant 
differences between the interests of mid-managers and those of staff. Mid-managers 
were significantly more interested in obtaining assistance in “performance indicators,” 
“benchmarks” and “development of survey items” than staff on an .05 level. The results 
also confirmed some basic differences in the responsibilities of mid-managers 
compared to staff. 

Table 1 presents the percentages of mid-managers, staff and total group who 
rated the topics as “Very Interested.” When 50% or more of the respondents rated a 
topic "Very Interested,” that percentage appears on Table 1 in bold print. This process 
helps narrow the number of topics to be considered for consultation or professional 
development activities. Therefore, based on this 50% criterion, mid-managers were 
very interested in assistance in 13 of the topics and staff in two of the topics. The top 
two topics for mid-managers were “design assessment project" and “identify what, 
when, who, how to measure.” The top two for staff were “develop quality checks” and 
“develop satisfaction surveys.” The top two for the total group were “data collection 
strategies” and “data interpretation.” 

Table 1 also demonstrates the importance of conducting a subgroup analysis, 
especially for heterogeneous groups. As noted above, if a needs assessment was 
conducted on only the total group the top two topics of the total group would not 
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accurately reflect the top needs of both mid-managers and staff. Additionally, if 
assessment needs of personnel were based solely on the percentages of the total 
group and not of mid-managers and staff separately, staff needs would be 
overshadowed by mid-manager needs since mid-manager percentages are generally 
higher than staff and therefore would be reflected in the percentages of the total group. 

Finally, topics with percentages lower than 50% may be worth addressing when 
the total number of respondents is large enough and the topic is important to the 
division or institution. For example, 48% of mid-managers and 47% of staff are very 
interested in “data use." The number of personnel amounts to 12 mid-managers and 20 
staff for a total of 32 personnel very interested in “data use.” With the current emphasis 
at this institution on planning and implementation based on data, such a topic may 
have more priority for consultation or personnel training. 

Technology and Electronic Searches 

Almost all (95%) student affairs personnel checked their e-mail at least once a 
day. More than half (55%) used Netscape Navigator as their primary browser. In regard 
to search engines, one-fourth of mid-managers used Excite or Yahoo and two-fifths of 
staff used Yahoo. This information is useful for future staff development activities 
involving Internet-based activities. 

Mid-managers and staff differed in opinion on their top concern in information 
technology. Mid-managers expressed concern for all seven items included in the 
survey with the top concern being the integration of information technology into 
program services; staff concern focused on adequate training in new software and 
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hardware. See Table 2. 

[Place Table 2 about here.] 

Mid-managers expressed greater interest in on-line searches than staff. Table 3 
demonstrates that mid-managers show a strong interest in performing efficient 
computerized searches and to a lesser extent in differentiating search engines and in 
performing searches for student assessment data. 

[Place Table 3 about here.] 

Data Analysis Method 

Half of all student affairs personnel used a statistical software package for data 
analysis. A higher percentage of mid-managers (56%) used statistical packages than 
staff (46%). Excel was used by both groups, whereby SPSS and SAS were used only 
by mid-managers. None used Lotus. See Table 4. 

[Place Table 4 about here.] 

In regard to content analysis, 20% of student affairs personnel performed 
content analysis using a software package. Both mid-managers and staff used Excel for 
this purpose rather than more complex packages, such as NUD-IST or Ethnograph. 

See Table 5. 

[Place Table 5 about here.] 

When asked if they would be very interested in an introductory workshop on 
statistical or content analysis, most student affairs professionals were interested in 
Excel or in SPSS. See Table 6. 

[Place Table 6 about here.] 
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Additionally, at this institution, student affairs has a resource room where SPSS 
is made available through a university license to student affairs personnel, including 
student assistants, who are working on research and assessment projects. When 
personnel were asked if they anticipate using this service in the next two years, one- 
third (34%) of them plan to use SPSS sometime within two years. 

Scheduling and Planning 

When asked about their time preference for workshops and meetings, student 
affairs personnel favored early morning (27%) or midmorning (27%). No one selected a 
late morning or evening period. Mid-manager time preferences were distributed 
throughout the day, but staff members definitely preferred an early or mid morning 
schedule. See Table 7. 

[Place Table 7 about here.] 

Mid-managers (40%) preferred small group meetings and staff (56%) preferred 
workshops for personnel development activities. See Table 8. 

[Place Table 8 about here.] 

Summary of Survey Results 

The assessment needs of student affairs mid-managers and professional staff a 
research university were identified using a web-based survey instrument. 

• Most student affairs professionals report a basic level of research and assessment 
expertise. 

• Although mid-managers and staff have common needs, each group has distinctive 
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interests. Mid-managers were more interested in accountability measures and the 
design of assessment projects. Staff interests favored more specific service- 
oriented measurement tools, such as satisfaction surveys and quality checks. 

• Student affairs personnel check their e-mail at least once a day and are positioned 
to respond to activities presented on the Internet. 

• In regard to technology, mid-managers are especially concerned about how to 
integrate information technology into program services, whereby staff are mainly 
concerned about adequate training in the use of new software and technology. 

• Mid-managers appear far more interested in on-line search topics than staff with 
the greatest interest being in performing efficient computerized searches. 

• About half of student affairs professionals use a statistical software package, and 
very few use content analysis software packages. Both groups favor the offering of 
introductory statistical analysis workshops in Excel and SPSS. 

• Mid-managers have no strong time schedule preference for training and 
development activities; staff prefer an early or mid morning schedule. 

• Mid-managers prefer small group meetings and staff prefer workshops for training 
and development activities. 

Problems and Possibilities Using a Web-Based Survey 
This study uses Perseus Survey Solutions for the Web (Perseus, 1998), one of 

several software packages for developing and administering electronic surveys via e- 

mail or the Web. The advantages of using such a package include its ease in use, 
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quality, support service, price of the package and cost effectiveness. Also, there are 
cost savings in paper, handling and postage. Survey Solutions does not require 
knowledge of html programming, and, for this particular project, serves as a model for 
student affairs practitioners who would like to utilize similar procedures in their 
research and assessment projects. Users can employ their institution’s or Perseus 
server to collect data. Results can be stored in various formats, such as an ASCII TSV 
(tab separated values) file on the server or Microsoft Access databases on a 
workstation. The package also includes analysis, charting and reporting functions. 
There are also no additional charges by Perseus for using their server or for the 
number of records processed. It is advantageous for student affairs research offices 
with little or no technical computer expertise to involve and collaborate with their 
university information technology office for needed assistance and consultation. 

Potential problems using web-based surveys include: lack of access to the 
Internet by respondents; lack of Internet use even with connectivity; minimal computer 
skills by respondents; lack of trust in confidentiality, anonymity and security; and, 
unexpected breakdown of equipment and software (Dillman, 2000; Schuh 2001). 
Additionally, subjects may become satiated with web-survey requests as its popularity 
grows. 



Implications for the Student Affairs Researcher 
The typical student affairs research office is a one-person office with a minimal 
budget and with a wide range of responsibilities (Malaney, 1999). Regardless of size, 
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most student affairs research offices provide consultation and training to its 
constituents. 

First, identifying the assessment needs of student affairs professionals help the 
researcher plan, prioritize and focus on the key consultation and training needs of the 
division. Thus, the researcher will have a plan to guide discussions and decisions, 
especially when challenged by multiple and competing requests for assistance by 
different audiences. Such a plan would better insure that the time and effort the 
researcher dedicates to consultation and personnel development activities are in line 
with the goals of the research office. 

Second, sub analyses of the needs assessment results can enable a researcher 
to identify and address collectively needs common, not only to mid-managers and staff, 
but to programs. For example, this study revealed that two of the larger programs, 
housing and student activities, were very interested in developing web-based 
instruments to survey students and to convert paper surveys to online surveys. 

Third, assessment results provide evidence that may be needed to advance 
proposals in support of student affairs research activities. 

Fourth, web-based surveys can be administered, processed and communicated 
efficiently. It can save the researcher time and effort. Conversely, with interest in and 
demand for online research increasing, the researcher will do well to continuously 
review priorities. 

Fifth, the web-based procedure used by the researcher can serve as a model for 
other student affairs professionals to utilize in their program planning and assessment 
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activities. To the extent that it does, it facilitates the goals of both parties. 

Sixth, web-enabled data collection will become an increasing part of the work style 
of all researchers independent of the size of the research office. 
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Table 1 

Mid-Managers and Staff Very Interested in Assessment Areas 



% Very Interested 



Assessment Areas 


Mid-Managers 

n=26 


Staff 

n=42 


Total Group 
n=68 


1 . Design assessment project 


67 


41 


53 


2. Identify what, when, who, how to measure 


67 


41 


53 


3. Data collection strategies 


60 


48 


54 


4. Data analysis 


58 


40 


48 


5. Data interpretation 


63 


47 


54 


6. Graph development 


44 


24 


33 


7. Report writing 


39 


31 


35 


8. Data dissemination 


35 


31 


33 


9. Data use 


48 


47 


47 


10. Surveys 


52 


41 


46 


1 1 . Student tracking 


54 


37 


44 


12. Focus groups 


38 


28 


32 


13. Telephone interviews 


29 


10 


19 


14. Observations 


42 


21 


30 


15. Document review 


38 


21 


28 
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16. Audio visual 


43 


30 


35 


17. Web survey forms 


54 


41 


46 


18. E-mail surveys and forms 


50 


41 


45 


19. Test scoring machines 


29 


18 


22 


20. Benchmarks 


52 


24 


37 


21. Performance indicators 


60 


35 


46 


22. Program evaluation 


60 


41 


50 


23. Accreditation 


33 


14 


23 


24. Council for the Assessment of Standards 


38 


23 


30 


25. Write measurable objectives 


56 


39 


47 


26. Measure use/demand/need 


52 


41 


46 


27. Measure effectiveness 


60 


43 


51 


28. Measure efficiency 


48 


40 


44 


29. Develop satisfaction surveys 


46 


50 


48 


30. Develop quality checks 


44 


52 


48 


31 . Develop survey items 


50 


38 


43 
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Table 2 

Information Technology Issues of Concern to Mid-Managers and Staff 



Information Technology Issues 


% 

Mid-Managers 


Staff 


Total 


1 . Integrating info tech into program services 


85 


45 


60 


2. Financing replacement of hardware & software 


58 


45 


50 


3. Financing the purchase of new technology 


69 


48 


56 


4. Providing adequate user support to students 


58 


41 


47 


5. Providing adequate user support to staff 


69 


55 


60 


6. Receiving adequate training to use new 


73 


62 


66 


software and technology 

7. Using the Web in on-line and distance ed services 


58 


41 


47 
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Table 3 

On-Line Search Topics of Interest to Mid-Managers and Staff 







% 




On-line Search Topics 


Mid-Manager 


Staff 


Total 


1 . Different search engines and their purpose 


50 


29 


37 


2. Performing efficient computerized searches 


69 


35 


50 


3. Web searches for student assessment data 


50 


41 


41 
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Table 4 



Statistical Software Packages Used bv Mid-Managers and Staff 



Statistical Software 
Packages 




% 




Mid-Managers 


Staff 


Total 


1. None 


44 


54 


50 


2. Excel 


28 


42 


36 


3. Lotus 


0 


0 


0 


4. SPSS 


16 


0 


7 


5. SAS 


8 


0 


4 


6. Other 


4 


3 


4 
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Table 5 



Content Analysis Software Packages Used by Mid-Managers and Staff 



Statistical Software 
Packages 




% 




Mid-Managers 


Staff 


Total 


1. None 


84 


77 


80 


2. Excel 


16 


20 


18 


3. NUD-IST 


0 


0 


0 


4. Ethnograph 


0 


0 


0 


5. Other 


0 


3 


2 
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Table 6 

Interest in an Introductory Workshop by Mid-Managers and Staff 



Interested in Introductory Workshops 


% Very Interested 
Mid-Managers Staff Total 


1 . Excel for statistical analysis 


44 


47 


45 


2. Lotus for statistical analysis 


18 


19 


18 


3. SAS for statistical analysis 


6 


29 


18 


4. SPSS for statistical analysis 


35 


44 


40 


5. NUD-IST for content analysis 


24 


23 


23 


6. Ethnograph for content analysis 


30 


13 


21 
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Table 7 



Best Time Schedule for Workshops for Mid-Managers and Staff 



Time Schedule 




% 




Mid-Managers 


Staff 


Total 


1. Early morning 


20 


31 


27 


2. Mid morning 


20 


31 


27 


3. Late morning 


0 


0 


0 


4. Around the noon hour 


20 


13 


15 


5. Early afternoon 


0 


6 


4 


6. Mid afternoon 


20 


13 


15 


7. Late afternoon 


20 


6 


12 


8. Evenings 


0 


0 


0 
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Table 8 



Preferred Mode of Personnel Development by Mid-Managers and Staff 



Preferred Mode 


% 

Mid-Managers Staff 


Total 


1 . One-to-one meetings 


16 


13 


14 


2. Small group meetings 


40 


22 


30 


3. Workshops 


28 


56 


44 


4. Telephone interactions 


8 


3 


5 


5. Electronic mail interactions 


4 


3 


4 


6. Literature 


4 


3 


4 
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